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Web-Based Continuous Outcomes 
Assessment: A Pilot Study 

Outcomes assessment and continuous 
improvement are essential elements of educa- 
tional programs. Emphasis on assessment is 
based primarily upon the requirements of 
accrediting agencies, but also on the perceived 
value of assessment in satisfying the demands 
for accountability in an increasingly competitive 
environment. Educational research has shown 
that measurement of educational outcomes can 
be used to inform an institution about educa- 
tional goals that are being satisfied and those 
that are not. This information can motivate and 


direct efforts to improve curricula. 

In practice, the measurement of educational 
outcomes is challenging and can be expensive. 
Results are often ambiguous or statistically 
unsound. The positive impact of continuous 
improvement on the curriculum is difficult to 
prove (Allen, Noel, & Rienzi, 2000). Despite 
the difficulties and expense, accrediting groups 
and other agencies have mandated outcome 
assessment and continuous improvement. Of 
special note are new criteria by the 
Accreditation Board for Engineering and 
Technology (ABET). The new criteria rely 


extensively on outcome assessment for accredi- 
tation, and there is a need for efficient and 
effective assessment processes to satisfy these 
requirements. These same outcomes are also 
applicable to many other programs, including 
those accredited by the National Association for 
Industrial Technology (NAIT), North Central 
Association of Colleges and Schools (NCA), 
and other accrediting groups. This article 
describes a pilot study of the implementation of 
a Web-based (TrueOutcomes©) assessment 
process in a technology program and examines 
the effectiveness of the process in terms of stu- 
dent and faculty buy-in and whether the process 
produces useful data and reports for accredita- 
tion and continuous improvement. 

We qualitatively measured the following: 

• Difficulty of learning the assessment 
process for instructors and students. 

• Quality of the descriptions of educational 
experiences submitted by students. 

• Ability of students to categorize their expe- 
riences according to educational outcomes. 

• Usefulness and appropriateness of the 
assessment process and reports. 

We quantitatively measured the level of 
participation of those students who participated. 

TrueOutcomes (formerly EnableOA) is a 
Web-based, software-driven outcomes assessment 
process that was designed to be consistent with 
the nine Principles of Good Practice for Assessing 
Student Learning, an online publication by the 
American Association for Higher Education 
(AAHE, n.d.), and the Program Evaluation 
Standards developed by The Joint Committee on 
Standards for Educational Evaluation (1994). 

The TrueOutcomes process (Walcerz, 

1999a, 1999b) collects both descriptions of 
educational experiences that instructors intend 
for their students and descriptions of educational 
experiences that students perceive they have 
received from their instructional experiences. 
Instructors prepare the former and students 
develop the latter. Every outcome description is 
associated with one or more of the educational 
outcomes developed by the instructor. Students 
are encouraged to attach electronic copies of 
their actual work (e.g., reports, PowerPoint 


presentations, CAD files, spreadsheets, 
programs, digital pictures of design projects, 
etc.) to support their descriptions. The outcome 
descriptions submitted by instructors are used to 
generate a matrix of coursework vs. educational 
outcomes that can be used in curriculum planning 
and evaluation. The descriptions that are submit- 
ted by students serve two primary purposes: 

1. Students develop and maintain a person- 
al electronic portfolio that serves as an 
extended resume and can be used for 
professional advancement. The useful- 
ness of this extended electronic resume 
is the primary motivation for students to 
participate in the assessment process. 

2. The descriptions are aggregated for a 
specific course or set of courses to see if 
students perceive and report the instruc- 
tor’s intended educational outcomes. 

The Department of Industrial Management 
at Southwest Missouri State University (SMSU) 
offers a baccalaureate degree in industrial man- 
agement (IM) with concentrations in construc- 
tion and manufacturing management. The IM 
program enrolls about 300 students. The fall 
2000 pilot study was conducted in two courses: 
Fundamentals of Engineering Drafting (TEC 
110) and Industrial Safety (TEC 250). In the 
spring 2001, courses in Statics and Materials 
Testing were also piloted. 

TEC 1 10 is a typical freshman-level draft- 
ing course in which CAD skills, sketching 
skills, and orthographic and isometric drawing 
techniques are practiced. TEC 250 is a sopho- 
more-level course that meets one evening each 
week for three hours and is populated mainly by 
young adults with full-time jobs. The course 
emphasizes management of occupational safety 
programs. 

Three accrediting agencies were associated 
with this study. Two directly impact SMSU and 
the third was included because of its impact on 
the development and application of the soft- 
ware. 

Those agencies and their impact are briefly 
described here. 
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NAIT accredits the SMSU industrial 
technology program and has the following 
requirements with respect to assessment that are 
taken from the industrial technology accredita- 
tion handbook: 

Assessment Plan and Integration: An 
assessment plan shall be comprised of, but 
not limited to, the following for each pro- 
gram: (1) program mission statement, (2) 
the desired program outcomes/student com- 
petencies, (3) evidence that the program 
incorporates these outcomes/student com- 
petencies, (4) the assessment measures used 
to evaluate 

student mastery of the student competen- 
cies stated, (5) compilation of the results of 
the assessment measures, and (6) evidence 
that these results are used to improve the 
program. (NAIT, 2000, §§ 5.16 & 6.16) 

NCA accredits SMSU as an institution and 
defines five criteria for accreditation and pat- 
terns of evidence to demonstrate the criteria. 
Statements relating to assessment are: 

Criterion 3: The institution is accomplish- 
ing its educational and other purposes. 

In determining appropriate patterns of 
evidence for this criterion, the Commission 
considers evidence such as: 

. . . assessment of appropriate student 
academic achievement in all its programs, 
documenting: proficiency in skills and 
competencies essential for all college-edu- 
cated adults; mastery of the level of knowl- 
edge appropriate to the degree granted; 
control by the institution's faculty of evalu- 
ation of student learning and granting of 
academic credit. 

Criterion 4: The institution can continue to 
accomplish its purposes and strengthen its 
educational effectiveness. In determining 
appropriate patterns of evidence for this cri- 
terion, the Commission considers evidence 
such as: 

. . . structured assessment processes that are 
continuous, that involve a variety of institu- 
tional constituencies, and that provide 


meaningful and useful information to the 

planning processes as well as to students, 

faculty, and administration. (NCA, 2000, p. 9) 

ABET has the following requirements with 
respect to assessment taken from criteria for 
accrediting engineering technology: 

Programs must have written goals that, as 
a minimum, focus on the student body served, 
employer expectations, resource allocation, and 
other factors affecting the program. Programs 
are required to have plans for continuous 
improvement and evidence that the results are 
applied to further development and improve- 
ment of the program. Each program is required 
to demonstrate achievements through various 
methods including student outcomes assess- 
ment and employer feedback. Typical evidence 
may consist of student portfolios including 
project work and activity based learning; 
results of integrated curricula experiences; 
nationally-normed subject content examina- 
tions; recent graduate surveys that demonstrate 
graduate satisfaction with employment includ- 
ing career development activities, mobility 
opportunities, and appropriate job title; and 
employer surveys that demonstrate satisfaction 
with recent graduates. Programs also must 
demonstrate that their graduates are readily 
accepted into the workforce and are prepared 
for continuing education. (ABET, 2000, 

Criteria 1 & 6) 

ABET developed and has introduced the 
following objectives as a part of Engineering 
Technology Criteria 2000 (ET2K). We believe 
that these objectives are compatible with the 
NAIT accreditation requirements. Since these 
outcomes were already developed and ready for 
use, they were adopted for this pilot study. The 
ABET objectives for graduates are: 

1. Demonstrate an appropriate mastery of 
the knowledge, techniques, skills, and 
modern tools of their disciplines. 

2. Apply current knowledge and adapt to 
emerging applications of mathematics, 
science, engineering, and technology. 

3. Conduct, analyze, and interpret experi- 
ments and apply experimental results to 
improve processes. 


4. Apply creativity in the design of sys- 
tems, components, or processes appro- 
priate to program objectives. 

5. Function effectively on teams. 

6. Identify, analyze, and solve technical 
problems. 

7. Communicate effectively. 

8. Recognize the need for and possess the 
ability to pursue lifelong learning. 

9. Understand professional, ethical, and 
social responsibilities. 

10. Recognize contemporary professional, 
societal, and global issues and are 
aware of and respect diversity. 

1 1 . Have a commitment to quality, timeli- 
ness, and continuous 
improvement. (ABET, 2002) 

In addition to NAIT and NCA, the state 
legislature, the State Coordinating Board for 
Higher Education, and the university adminis- 
tration all encourage verifiable assessment. 

Implementation 

Outcomes Assessment Solutions (formerly 
Enable Technologies), an application service 
provider, established an application Web site for 
SMSU on their Web server. Therefore, the insti- 
tution did not need to buy hardware or software. 
Outcomes Assessment Solutions was provided 
with a spreadsheet containing the names of the 
courses, the instructor, and rosters of students 
participating in the pilot study. They then 
imported this information into the software to 
establish accounts for all the involved students 
and the instructor. The 1 1 pre-existing educa- 
tional outcomes defined by ABET ET2K were 
also entered into the software. 

Instructor intentions were composed for the 
fall courses, in consultation with Dr. Walcerz, 
the service provider representative. In the spring 
course, intentions were added independently by 
the instructor. The objectives intended for the 
students taking the courses were compared with 
the previously listed ABET-TAC standards. This 
exercise was found to be helpful in better defining 
course objectives. This process exposed intended 
outcomes that had not been well formulated and 
also revealed that course content was much 
broader in scope than was initially perceived. 


Examples include: 

• Intended outcome: Students will learn to 
utilize computer aided design (CAD) 
software. This matches Outcome 1 : 
“Demonstrate an appropriate mastery of 
the knowledge, techniques, skills, and 
modern tools of their disciplines.” 

• Intended outcome: Students will develop 
sketching skills, which addresses this 
same standard. 

• Intended outcome: Students will work in 
small groups to check each other’s 
drawings before final submission. This 
matches Outcome 5: “Function effectively 
in teams.” 

The lack of submissions for last outcome 
emphasized that specific instruction in team 
building is needed for teamwork to be an outcome. 

TEC 250, the industrial safety course, 
addressed different standards. In one instance 
we were able to take advantage of the global 
safety officer for General Electric fractional 
horsepower motor plants, who had just returned 
from a visit to a new manufacturing plant in 
India. We learned from her that their method for 
transferring concrete is significantly different 
than the concrete pumps we now find so famil- 
iar in the United States. In India women were 
engaged to transfer concrete by climbing lad- 
ders carrying the concrete, balanced in baskets, 
on their heads. This unintended consequence 
addressed the ET2K Outcome 8: “Recognize 
contemporary professional, societal, and global 
issues and are aware of and respect diversity.” 

Working in groups to develop reports and 
requiring many brief written reaction papers 
addresses Outcome 7, “Communicate effective- 
ly,” as well as Outcome 5, “Function effectively 
on teams.” Communication with Blackboard 
software and associated e-mail techniques is 
another example accomplishing Outcome 1, 
“Mastering modern tools of their disciplines.” 

The process of working through course 
objectives and comparing them to the outcomes 
yielded a much better appreciation of how the 
objectives fit into the overall scheme of devel- 
oping, to use the campus vernacular, “an edu- 
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cated person.” The process of formulating these 
objectives and associating them with the ET2K 
standards initially took about one hour for each 
course. This involved reviewing the course 
syllabus and text(s) to identify what reasonable 
outcome objectives might be, creating instructor 
intended outcomes (word processing software 
recommended), and copying outcomes into the 
TrueOutcomes software. 

Collecting Student Data 

We presented the assessment software to 
students in two different ways. Because the TEC 
110 class is a combination lab/lecture format, 
we were able to present the software to students 
as a laboratory exercise. After the first major 
test, the students were provided a set of instruc- 
tions that directed them through help menus in 
the TrueOutcomes software. The students 
reported that these instructions were easy to 
follow and, after reading the introductory mate- 
rial, proceeded to make entries. Later analysis 
revealed that they were not all successful in 
making entries. 

The same instructions used for TEC 110 
were distributed to the TEC 250 class via e-mail 
and the Internet using Blackboard software. Later 
discussion with an informal sampling of students 
indicated that the TrueOutcomes “Help” process 
was easy to follow. However, analysis with the 
software indicated that only a little over 50% of 
the students successfully submitted entries. 
Speculation is that some students had failed to 
complete the submission process by missing a 
common last step . . . clicking the Submit Button. 

Student Participation in the 
Assessment Process 

A total of 37 students participated in the 
fall assessment process in two courses. 
Seventeen students submitted from one to three 
experiences to their electronic portfolios. In the 
software a color bar chart. Student Responses, 
provides data on student participation and indi- 
cates the percentage of students submitting 
experiences as well as the number submitted. In 
general, about one third of the students did not 
participate at all; one half submitted a single 
experience, and one sixth submitted multiple 
experiences. The apparent lack of participation 
may have been related to not clicking on the 


Submit Button as entries were made. This theo- 
ry is reinforced by the observation that the TEC 
110 students who were observed in a lab 
exercise had lower overall participation than 
the TEC 250 class. 

Quality Analysis of Student Narratives 

In order to participate in the assessment 
process, students had to compose narrative 
statements of their classroom experiences, 
focusing on concrete descriptions of what they 
had done rather than conclusions about what 
they had learned. This cognitive engagement in 
the evaluation process is one of the most valu- 
able attributes of this assessment system. A total 
of 26 narratives were submitted in the fall and 
analyzed for quality. Good narratives were 
written in first person, described the student’s 
work in good detail, and dealt with a single 
experience or a group of thematically related 
experiences. An example: 

... a trip to the . . . plant and I got to 
observe first hand . . . safety and health pro- 
cedures on the job. . . . this trip it gave me 
a[n] understanding on what should be done 
in order to keep the company you work for 
safe, and this is what this class is all about 
understanding what it takes to keep the com- 
pany you work for safe, so you don't have 
lost work days, have to pay workerman's 
[sic] comp, hire a replacement . . . and pay 
for overtime to catch up for the absents [sic] 
of a[n] employee. You need to have an 
understanding of safety and health, on the 
job, or it could cost the company a lot of 
money in hidden costs. With the tour ... I 
saw a company with good safety procedures, 
in the employee's working environments . . . 

Moderate narratives were written in first 
person, did not have enough detail, and 
sometimes included a collection of unrelated 
experiences. For example: 

Recently we took a tour of a local manufac- 
ture [r]. I thought the trip was a good idea 
and very informative. I It [sic] was well 
worth the time to do so. It would be great if 
more such trips were possible. 

Poor narratives were either too short (e.g., a 


single 3-word phrase) or talked about what the 
student learned instead of what he or she did. 
Example: 

I feel that TEC 250 is a valuable course and 
all industrial technology majors should take 
it or something close to it. Safety should be 
a huge part in the industry, however it is not 
always a main priority. I think all managers 
should be trained in the safety arena. 

An analysis of the students’ narratives 
revealed 11 good quality narratives, 10 moder- 
ate narratives, and 4 poor narratives. The 
quality analysis of student narratives showed a 
substantial variation in quality between courses: 
TEC 250 had nearly 70% good quality narra- 
tives, while the TEC 110 class had nearly 70% 
medium to poor narratives. TEC 250 had more 
nontraditional students; therefore, student 
maturity would be expected to influence 
narrative quality. The software also provides a 
graphical display of the quality analysis. 

For every experience students submitted, 
they selected the educational outcome that 
reflected their individual narrative. Our analysis 
showed that a majority of students selected too 
many outcomes; in other words, students 
selected outcomes such as “an ability to func- 
tion on teams” when their narrative contained 
no mention of teamwork at all. A graphic show- 
ing Response Appropriateness is also provided. 
We found nearly 70% selected extraneous 
outcomes. Clearly more training in the use of 
the system is indicated. We believe this is con- 
sistent with the customary “mark sense” course 
evaluations where no active thought processes 
might be engaged. The cognitive engagement 
potential made available with this system could 
clearly provide some benefit in assessment. 

The TrueOutcomes software automatically 
generates assessment reports based on the 
descriptions of experiences that students and 
instructors submit. A bar-chart presentation is 
available showing the percentage of students 
submitting experiences related to each of the 1 1 
outcomes and the number of experiences related 
to that outcome. According to the data analysis, 
the percentage of students who perceived and 
reported at least one significant educational 


experience demonstrating “an ability to commu- 
nicate effectively,” “an ability to function on 
teams,” “an understanding of professional, ethi- 
cal, and social responsibility,” and “an ability to 
identify, analyze, and solve technical problems” 
was 53%, 42%, 42%, and 37%, respectively. If 
we only consider the students who participated 
in the assessment process, the percentages are 
77%, 62%, 62%, and 54%, respectively. If the 
quality of the student narratives was moderate 
or good and the students were able to select 
appropriate outcomes, then it is patently clear 
evidence that TEC 250 is developing those four 
outcomes to a substantial degree and other 
outcomes to a lesser degree. 

The software also allows an instructor to 
“drill into” any of the outcomes to read the stu- 
dent narratives, review the selected outcomes, 
view attached documents, and thus to check the 
validity of the data. As discussed previously, 
the quality of narratives in TEC 250 was quite 
good, but the selection of outcomes often 
included more than the narrative justified. 
However, the raw evidence, the student narra- 
tives and attached documentation, is available to 
the instructor for assessment purposes. 

A standard matrix of technology courses vs. 
the outcomes they are designed to develop is 
also available. Across the top of the matrix are 
the 1 1 educational outcomes. Down the first 
column are listed all of the technology courses 
offered. The cells of the matrix are either blank, 
which means that the instructor does not intend 
to develop the associated outcome in that 
course, or else a 100%, which means that all of 
the students in the associated course have an 
instructor intending to develop the associated 
outcome. TrueOutcomes uses percentages 
instead of the more common “Xs” because 
some courses are not uniform across all sections 
and instructors. For example, one section may 
have an instructor who uses project teams, thus 
developing teamwork, while another section of 
the same course may have an instructor who 
does not. If there are 60 students in the former 
section and 40 in the latter, TrueOutcomes will 
report 60% in the cell associated with teamwork 
and that course. 

When comparing experiences submitted by 
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students with experiences intended by the 
instructor, it was clear that the instructor’s 
intentions were fulfilled with respect to the 
outcomes relating to teamwork, communication, 
and professional responsibility. The instructor’s 
intentions were partially fulfilled with respect to 
outcomes relating to knowledge and tools, and 
commitment to quality. The intended outcomes 
of global issues and lifelong learning were not 
being well met. It is interesting to see that a 
large percentage of students reported developing 
problem solving, when this was not an intended 
outcome of the course. This initial analysis 
must be tempered by the knowledge that most 
students selected extraneous outcomes, so 
“drilling into” the reported experiences will be 
necessary to verify the results until students 
become more familiar with the process. 

The ability to see what was happening as 
the term progresses is most intriguing. This 
“real time” access to student perceptions allows 
adjustments with current students as opposed to 
the standard end of term evaluation. This is 
preferable to having input that can only be 
applied to the next group of students. The fact 
that students have to be cognitively involved 
with this process also provides much better 
insight into evaluations. 


Conclusions 

The software and process were straightfor- 
ward to learn and use for both the instructor and 
students. Student participation and the quality 
of student submissions were acceptable but not 
as good as desired. Analytical reports provide 
meaningful feedback and can be used to docu- 
ment the continuous improvement process for 
accreditation purposes. The process does not 
require an excessive time commitment on the 
part of the instructor. Students can access the 
software at their convenience in addition to the 
creation of a portfolio of personal achievements 
that may be of value in future endeavors. 

The effort required to set up the 
TrueOutcomes program is balanced by the pro- 
vision of in-depth insight into the educational 
process. That insight, into the way that activities 
engage students and address outcomes, provides 
new opportunity by emphasizing areas where 
improvements can be made. 
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